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Abstract.  The  two  sided  Lanczos  algorithm  is  known  to  suffer  from  serious 
breakdowns.  These  occur  when  the  associated  moment  matrix  does  not  permit 
triangular  factorization.  We  modify  the  algorithm  slightly  so  that  it 
corresponds  to  using  a  1^1  "pivot"  in  triangular  factorization  whenever 
a  1x1  pivot  would  be  dangerous.  The  incidence  of  breakdown  is  greatly 
reduced.  The  price  paid  is  that  the  tridiagonal  matrix  produced  by  the 
algorithm  now  has  bumps  whenever  a  2x2  pivot  is  used.  .• 
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1 .  THE  LANCZOS  ALGORITHM  AND  ITS  BREAKDOWN 


The  most  popular  way  to  obtain  all  the  eigenvalues  of  a  nonsymmetric 
n  X  n  matrix  B  is  to  use  the  QR  algorithm  which  is  readily  available 
in  most  computing  centers.  As  the  order  n  increases  above  100  the  QR 
algorithm  becomes  less  and  less  attractive,  especially  if  only  a  few  of  the 
eigenvalues  are  wanted.  This  is  where  the  Lanczos  algorithm  comes  into  the 
picture.  It  does  not  alter  B  at  all  but  constructs  a  tridiagonal  matrix 
J  gradually  by  adding  a  row  and  column  at  each  step.  After  several  steps 
some  of  the  eigenvalues  9^  of  J  will  be  close  to  some  eigenvalues  A|^ 
of  B  and  by  the  n^^  step,  if  nothing  goes  wrong,  9^.  =  ,  i  =  l,...,n. 

This  description  is  correct  in  the  context  of  exact  arithmetic.  Unfortunately 
things  can  go  wrong,  even  in  the  absence  of  rounding  errors.  For  the 
relation  between  these  troubles  and  orthogonal  polynomials  see  [2]. 

In  order  to  discuss  these  troubles  we  must  give  some  details  about 
the  algorithm.  Let  J|^  be  the  k  x  k  tridiagonal  produced  at  step  k 
of  the  algorithm.  There  are  infinitely  many  tridiagonal  matrices  similar 
to  B  and  is  one  of  them.  Thus  for  some  matrix  Q^=  (q.|,...,q^)  we 
have 

Q„-’bq„  =  j„  (1) 


It  simplifies  the  exposition  considerably  to  introduce  a  redundant  symbol 
and  write  P^*  instead  of  .  The  superscript  *  indicates  conjugate 

transpose. 

Let  i  (p.|,...,p^)  and  replace  (!)  by  two  separate  relations 


Pn*Qn  =  I  • 

Pn*BQ  =  . 


(2) 

(3) 
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We  mention  in  passing  that  when  B*  =  8  =  A  then  we  can  arrange 

that  =  Q^.  The  difficulty  we  are  going  to  describe  cannot  occur 

when  =  Q„. 
n  n 

By  equating  elements  on  each  side  of  BQ  =  and  P*  B  =  J„P* 

■'  'I  ^  ^n  nn  nnn 

in  the  natural  increasing  order  we  shall  see  that  B,  and 
essentially  determine  all  the  other  elements  of  P„,  Q  ,  and  .  On 

writing 

“1  ^2 
^2  ''^3 

62 

• 

we  find 

*Bqi  =  . 

and 

Bq^  =  +  q2S2  '  "  “iPl*  ’>'2^2*  * 

Hence  q2  and  P2*  are,  respectively,  multiples  of  "residual"  vectors 
r^  =  Bq^  -  q^a^  .  s^*  H  p^*B^  -  a^p^*  . 

Furthermore,  since  P2*<l2  ~  ^  (2),  we  have 

Si*'"i  “  T2P2*^2^2  ”  ''^2^2  ~  ***2’ 
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If  ^2^0  and  is  given  any  nonzero  value  then  y2»  ^2’  ^Z* 

are  all  determined  uniquely.  A  good  choice  is  32  =  • 

The  general  pattern  emerges  at  the  next  step,  on  equating  the  second 

columns  on  each  side  of  BQ  =  Q  J  and  P*„B  =  J  P*  , 

n  n  n  n  n  n 

P2*Bq2  =  0.2, 


^^2  ^1^2  ^2^2  ^3^3*  ^2*^  ~  ^2^1*  ^2^2*  "'^3^3*' 

At  this  point  we  can  compute  the  "residual"  vectors 

^2  ”  ®^2  ”  ^1*^2  *  ^2^2*  ^2*  ~  P2*®  ”  ^2^1*  "*  ^2^2* 

and 


0)3  =  S2*r2  =  "'^3^3  • 

If  /  0  and  B3  is  given  any  nonzero  value  then  Y3.  P3.  and  P3* 
are  all  determined  uniquely.  And  so  it  goes  on  until  some  vanishes. 

This  is  the  Lanczos  algorithm.  It  must  terminate  at  the  n^^  step  with 
0)^^^  =  0  but  it  may  stop  sooner. 

Premature  termination  at  say  step  j  (<  n)  can  occur  in  two  ways: 

(I)  either  r.  =  0  or  s.*  =  0*  or  both, 

J  J 

or 

(II)  r^  /  0,  $.*  f  0*,  but  0)^^^  =  0. 


liiliMllliliiiiWiM 
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In  the  1950's  when  the  Lanczos  algorithm  was  regarded  as  a  way  to 

compute  Case  I  was  regarded  as  a  mild  nuisance.  If  r^  =  0  then 

any  nonzero  vector  orthogonal  to  can  be  chosen  as  Qj+i  • 

Similarly  s^  =  0  gives  ample  choice  for  Pj+i • 

Today,  regarding  Lanczos  as  a  way  to  find  a  few  eigenvalues  of  large 

B  it  seems  better  to  stop  at  Case  I  in  the  knowledge  that  every  eigenvalue 

of  J.  is  an  eigenvalue  of  B.  If  more  eigenvalues  are  wanted  then  it  is 
J 

best  to  start  the  Lanczos  algorithm  afresh  with  new,  carefully  chosen 
starting  vectors  q.|  and  p.|  . 

The  real  trouble,  which  cannot  occur  when  B  =  B*  =  A,  is  Case  II. 
Wilkonson  calls  this  a  serious  breakdown.  There  seems  to  be  no  choice  but 
to  start  again  but  no  one  has  been  able  to  suggest  a  practical  way  to 
choose  the  new  q.|  and  p.|*  so  as  to  avoid  another  wasted  run  of  the 
algorithm.  That  is  why  the  Lanczos  method  has  not  been  used  much  when 
B*  B .  In  this  article  we  propose  a  modification  of  tne  algorithm  which  great ly 

reduces  the  occurrence  of  Case  II.  The  price  paid  for  this  convenience 
is  that  J  is  not  quite  tridiagonal.  There  is  a  small  bump  (or  bulge)  in 
the  tridiagonal  form  to  mark  each  occurrence  (or  near  occurrence)  of 
Case  II. 


Example  1 .  (No  breakdown) 

B  =  diag(2,3,4),  q^*  =  ^1,1,1),  p^*  =  ^(1,2,1) 
Stsp  1  I  ot^  “  3)  (U2  “  82  “  ^ »  Y2  ~  "2* 

q^*  =  ^(-1,0,1),  P2*  =  (-1,0,1). 

St6p  2 1  ct2  “  3  J  ^3  *  ^3 

q3*  =  ^1,-1,1),  P3*  =  ^1,-2,1). 
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Step  3:  ^2  =  3,  =  0. 


2.  THE  TWO  SIDED  GRAM-SCHMIDT  PROCESS. 

The  serious  breakdown  described  above  is  not  limited  to  the  Lanczos 
algorithm.  It  can  occur  in  any  attempt  to  use  the  familiar  Gram-Schmidt 
process  to  produce  a  biorthogonal  (or  biorthonormal)  pair  of  sequences. 
Our  modification  of  Lanczos  seems  more  natural  in  such  a  context. 

Let  F  =  (fi,...,f^)  and  G  =  (gi»....gp|)  be  given  real  nonsingular 

n  by  n  matrices.  In  other  words  {f^-.-.f,^}  and  {g^ . g^}  are  each 

a  basis  for  the  vector  space  of  column  n-vectors.  We  want  to  produce 
a  new  pair  of  bases  {qi,...,q^}  and  such  that 

V^n  =  ^n  =  diag(co^,...,(.J 

and,  for  each  j  =  1 , . . .  ,n 

span  {q.|,...,q^}  =  span  {f.j ,fj}, 
span  {p.|,...,pj}  =  span  {g., , . . .  .gjl . 

The  Gram-Schmidt  process  dictates  that  at  step  j 
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All  goes  well  until  oj.  =  0  for  some  j.  This  can  happen  despite  the 
fact  that  F  and  G  are  nonsingular. 


Example  2. 


1  1  1 


1  1  2 


1  1  1 


F  =  1  2  1  ,  G  =  0  0  1 


0  1  0 


Step  1  ;  q.|  -  f-j  ,  p.|  =  g.j  ,  Lij  =  1  . 

Step  2:  q2*  =  (0,1  ,0),  P2*  =  (-1.0,1),  CO2 


One  remedy  for  the  case  ijj.  =  0  is  quite  natural.  If  p.  f  0  then 

J  J 

recompute  q^  using  »  in  place  of  f  ^ .  If  this  fails  too  then  try 
fj^2«  ^nd  so  on.  If  F  is  nonsingular  there  must  be  some  i  >  0  such 
that  will  yield  a  nonzero  value  for  . 

j  ^  \ 

Here  is  a  formal  proof  of  tne  last  remarxs.  Let  q.'  ’  aenote  tne 

J 

vector  obtained  by  using  f,  instead  of  f.,  i.e., 

J 


W  .  f,  .  E 


k  i=l 


Lemma:  If  p.  ^  0  and  p.*q.'  =  0  for  k  =  j,j+l,...,n  then  F  is 

W  J  J 

singular. 

Proof:  By  construction  =  0  Tor  i  <  j. 

Hence  p^  J_  span(f.| , . , .  ,f^_.j ) . 

Thus  0  =  =  P,.*f.  -  0,  for  all  k  >  j  . 

Thus  Pj  J_  span(f^,...,f^). 

Thus  p.*F  =  0  and  F  is  singular. 

J 


If  the  modification  succeeds  the  first  time  then  only  one  property  of 
Gram-Schmidt  has  been  sacrificed,  namely 
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span  (q^,...,q^)  f  span  . 

Moreover,  if  no  further  breakdown  occurs  then 

span  )  =  span  (fp...,f^)  for  i  >  j. 

In  many  applications  this  price  is  well  worth  paying. 

The  Lanczos  algorithm  can  be  regarded  as  the  two  sided  Gram-Schmidt 
process  applied  to  the  columns  of  the  special  matrices 

K  =  E  (q^,Bq^.3^g^....,B""''g^), 

and 

K*  =  K^*  E  (p^*,p^*B,p.,*B^...,p*B"‘^  . 

We  will  not  establish  this  result  but  content  ourselves  with  stating  the 
key  observation,  namely 

Span  (q.|  ,q2, . .  .,qj,Bqj)  =  Span  (q^ , . . .  ,qj  ,B'^q^ )  . 

The  K  matrices  are  called  Krylov  matrices  and  the  pleasant  fact  is  that 
most  of  the  work  required  for  general  Gram-Schmidt  disappears  in  this  case 
because 

p.*Bq.  =  0  for  i  <  j-1  . 

1  3 


Thus  the  general  formula 
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“ijti  ■  -  ,1; 

collapses  to 

=  Bq.  -  q.a.  -  q..^3._^. 

3.  TRIANGULAR  FACTORIZATION  OF  MOMENT  MATRICES 

Consider  again  the  matrices  K  and  K*  defined  in  the  previous  section. 
Note  that  the  (i.j)  element  of  K*K  is  ( p.|*B’ ) ,  so 

i<*K  =  M  =  M(p^,q^);  =  p^*B'^Jq^. 

In  order  to  use  this  observation  we  need  some  basic  facts  about  the 
Lanczos  algorithm  (see  [4],  [5],  or  [7]).  If  it  does  not  breakdown  it  oro- 
duces  matrices  P  and  Q  such  that 

Pj*,  ■  Xj(B)<i,/(  3,). 

Pj+i  ■  xj<B*)P,/(  Y,). 


where 


-  (t  -  aj)Xj(t)  -  a)jXj_i(T)  , 

and  Xi  is  the  characteristic  polynomial  of  the  tridiagonal  matrix  J.. 
4  J 

In  other  words,  for  each  j  ,q  is  a  linear  combination  of  the  first  j 
columns  of  K  while  Pj  is  the  same  linear  combination  of  the  columns 
of  K,  up  to  scaling.  This  can  be  expressed  compactly  in  matrix 
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notation  as 

Q  =  KL‘*n‘^ , 
P  =  KL'*ii‘^  . 


(4) 


Here 


n  =  diag(l  ,  ...  ) , 

ii  =  diag(l  ,y^,y^r^,  ...)  , 

and  L  is  the  unit  lower  triangular  matrix  such  that  L"*  =  (L'^)*  has 

the  coefficients  of  x-  above  the  diagonal  in  the  column. 

J 

Using  (4)  we  can  rewrite  I  =  P*0  as 
I  =  P*Q  =  (:‘k'^K*)(KL'*n*^) 

i  .e. 

M  =  Lrl*  ,  (5) 

where 

il  “  ITII  ~  diag(l> >  •  •  *  > )  ■ 

This  result  is  not  new  (see  Householder,  [2  ])  but  it  is  worth 
emphasizing  that  the  product  0)2-. -ui^.  is  the  ith  pivot  which  arises  in 
performing  Gaussian  elimination  on  the  moment  matrix  M  associated  with 
B,  and  p^ . 

When  B  ^  B*  the  moment  matrix  M  need  not  be  positive  definite 
and  so  triangular  factorization  need  not  be  stable,  even  when  M  is 
nonsingular. 


t  w.ii 


. _ _ 


The  best  known  remedy  for  instability  is  to  use  some  form  of  row  or 
column  interchange  whenever  an  ui.  is  too  small.  The  standard  "partial 
pivoting"  strategy  is  not  available  because  a  whole  column  of  M  is  not 
known  in  the  middle  of  the  Lanczos  process.  An  alternative  remedy  is  to 
enlarge  the  notion  of  a  "pivot"  to  include  2  2,  or  even  larger  sub¬ 

matrices.  This  idea  is  discussed  in  Parlett  and  Bunch,  1971,  [4].  It  is 
the  basis  of  our  method.  Whenever  a  2x2  pivot  is  used  the  tridiagonal 
J  bulges  outwards  temporarily. 

In  the  context  of  the  Lanczos  process  our  remedy  for  a  tiny 

J 

requires  us  to  compute  q^^.|  at  the  same  time  as  q^,  and  p^.^.|  at  the 
same  time  as  p ^ .  The  formulas  for  these  "Lanczos"  vectors  are  somewhat 
different  from  the  standard  ones.  We  call  our  modification  the  "look-ahead 
Lanczos  algorithm"  because  it  computes  at  the  current  step  some  quantities 
wnich  are  not  usually  needed  until  the  next  step  in  the  standard  Lanczos 
process . 


4.  THE  NEXT  PIVOT 

The  decomposition  M  =  L.lL*  is  never  found  explicitly.  In  order  to 
make  use  of  the  idea  of  using  a  2x2  pivot  it  is  necessary  to  determine 
the  top  left  2x2  submatrix  of  what  would  be  the  reduced  matrix  in  the 
trangular  factorization  of  M.  These  three  numbers  can  be  determined  from 
the  information  available  in  the  Lanczos  process. 

After  i-1  steps  of  the  standard  algorithm  we  have 


r, .  -  Bq^_^  -  (B)q^/{B2- • -Bi.-i )  . 

s. *  =  p*._.,B  -  ot._^p*._^  -  =  Pi*Xi.i(B)/(Y2--->i-l 


), 


Instead  of  normalizing  r^  and  s.* 


to  get  q^  and  p^*  we  can  look 
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ahead  not  to  the  next  Lanczos  vectors  but  to  any  vectors 

^1+1* 

the  plane  (r.,r.^^)  =  the  plane  (q. 

the  plane  (s^.*,s^.^^*)  =  the  plane  (p^-*.Pt+i*)  • 


The  simplest  choice  for  r^^^  and  is 


-=  “''i  -  '’1-1  -‘I- 
=  =“  =1*6  -  "1^1-1*  ■ 

The  coefficient  oj^.  ensures  that  is  orthogonal  to  all  the  known 

p's,  namely  p^ , . . .  ,p^.  ,  and  also  that  s^.^.|  is  orthogonal  to  q.|,...,q^. 

Note  that  if  we  choose  as  q^.  arty  vector  in  the  plane  (i"^  ) 

other  than  a  multiple  of  r.  then  q^  will  be  of  the  form 

=  'i^(B)q.,/(62--.Si) 

with  ^  a  monic  polynomial  of  degree  i  instead  of  the  usual  X-j  ]  • 
Moreover  it  will  be  possible  to  choose  q^^l  to  be  of  the  same  form,  using 
a  different  tp  but  of  the  same  degree  i.  This  is  a  mild  generalization 
of  the  basic  Lanczos  algorithm.  The  degrees  of  the  new  Lanczos  polynomials 
are  still  nondecreasing  but  they  do  not  always  go  up  by  one  at  each  step. 
Before  making  such  a  choice  we  compute 


irJI.IIs^.  II,  and  cos  Z.  (r^.,s..)  =  r^. !sT 
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‘“i+1  "  ^+l*'"i+l’  "  S*+/i  ’ 

“"^i+l ■i-^'^i+1’  ^i+1)  =  !'^i+i !/  '"i+r'"  ^i+1 

Our  choice  of  ,  etc.  must  be  based  on  the  matrix 


'^1+1 


It  turns  out  that  the  top  left  2x2  submatrix  of  the  reduced  matrix 
in  the  associated  triangular  factorization  of  M  is  (0)2- 1^'^i’ 
but  no  use  will  be  made  of  this  fact. 

If  both  .j.j  =  0  2nd  W^.  is  singular  then  more  drastic  measures  are  needed 
to  salvage  the  algorithm.  We  will  not  pursue  this  case  here.  When  =  0 
then  the  standard  Lanczos  algorithm  breaks  down.  Yet  if  is  invertible 
it  is  easy  to  choose  suitable  q's  and  p's  so  that  the  algorithm  can 
proceed . 

The  equations  above  can  be  condensed  into  block  form. 


(r^.r.^l)  =  B(q._.,,r.)  -  q)  '  ‘’i-2(>i-l  • 


cti  1  0 


^®i’^i+l^*  ""  ^^i-l ■  (  u).  0  )  ^^i-l ’^i  ^*  ■(  0  )  Pi-2*’ 


Various  factorizations  of  yield  various  selections  for  new  q's  and 
p's.  These  are  discussed  below.  We  write  di  for  u).j+i  and  9  for  8.j 


Let  us  drop  the  subscript  i  and  write 

W  =  VU. 


1 .  LU  factori zation 


/  1  0\  /w  0 

V  =1  I  ,  U  =  I  2 


V0/CiJ  1 


,0  (i-0  /a)> 


2.  UL  factorization 


\  0  w/  Ve/w  1 


3 .  QR  factorization 


'T^  9(u+co) 
0  (axo-0^) 


-1  _  ,  2.,2^+  1/2 
T  ,  T  =  (O)  +0  J 


4  .  m  with  interchange 


/  Oj/  0  1  \  /  0  0)  i 

V  =(  •  = 

\  1  0  /  \o  Q-mo/QJ 


5.  THE  SMALLEST  ANGLE 

In  order  to  determine  the  smallest  angle  between  the  planes 
and  orthonormal ize  the  bases.  Let  the  result  be 

(F^.f^+l)  and 

when  the  Gram-Schmidt  process  is  used.  The  (matrix)  angle  between  the  two 
planes,  call  it  4>,  is  defined,  see  [1]»  by 
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cos  $  =  (s..g.^^)*(r.,f.^,). 

Let  the  Singular  Value  Decomposition  of  cos  ^  be 

cos  <1>  =  U  Z  V* 

where  Z  =  cliag(a^  .02^  ^ind  ^  appropriate  definition  of  new 

Lanczos  vectors  to  ensure  the  smallest  L(q.  ,p^)  is 

(Pj.Pi*,)  ■  (?,,3,^,)u*r-’'^ 

Comments .  No.  1  corresponds  to  the  standard  Lanczos  algorithm.  No.  2 
corresponds  to  simply  swapping  s^*  with  s^*B  and  r^.  with  Br..  in  the 
Lanczos  algorithm.  One  consequence  is  that  the  J  matrix  will  bulge  out 
of  tridiagonal  form  on  both  sides.  No.  3  is  always  stable  and  keeps  the 
bulge  on  one  side.  The  same  advantage  accrues  from  No.  4  and,  as  might 
be  expected,  is  somewhat  simpler  than  No.  3. 

We  want  to  use  No.  1  whenever  this  is  a  reasonable  strategy  but 
when  0)  is  zero  or  very  small  it  is  vital  to  choose  one  of  the  other 
procedures.  We  have  tentatively  chosen  No.  4  on  the  grounds  of  simplicity. 
The  interesting  question  which  we  now  address  is  when  to  use  No.  4. 

Initial  success  with  No.  4  has  not  encouraged  us  to  implement  No.  5. 

Criterion  for  Choosing  a  2x2  Pivot 

When  Factorization  No.  4  is  chosen  then 
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lo  J  ■ 

(Pj.P,*,)  ■  (s,,,,s,)  /l  -»(/6A 

V  j  ■ 

to  within  scalar  multiples.  Note  the  interchange.  Hence 


cos  ,  (q^,p.)  =  COS  ,  (r^-.s-^^)  = 

(9.-a3.ai.^^/e.) 

COS  ^  (qi+vPi+i)  - - 

'^i+r/‘^i+i\  ""i  ■  ^i‘/!!iVl+l 

V  e,.  ) 


Both  these  numbers  are  readily  computable,  without  forming  the  new  vectors, 
provided  that  r^.^^*r^.  and  s^^|*s.  are  known.  The  choice  between  No.  1 
and  No.  4  reduces  to  a  comparison  of 


=  jcos  ^(r.  ,s.)|  and  412  =  '"in  tU,- 1 . 1'4'i+i  1  >  • 

If  ({i.|  <  lOOe  and  then  our  algorithm  stops  and  reports  failure. 

Otherwise,  for  a  given  bias  factor  we  proceed  as  follows:  if  4)^  ^  (bias 
factor)  4)2  then  use  standard  Lanczos  else  use  Factorization  No.  4.  When 
bias  =  0  we  recover  the  standard  algorithm.  Currently  bias  =  1. 

Sometimes  the  test  declares  that  a  standard  Lanczos  step  is  safe.  In 
such  cases  4).|  and  4)2  ane  not  used  and  their  computation  may  be  regarded 
as  an  overhead  of  4  inner  products.  No  matrix-vector  products  are  "wasted". 
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The  4  inner  products  arise  as  follows.  We  need 


We  may  regard  the  second  and  third  terms  on  the  right  hand  sides  as  the  price 
to  be  paid  for  knowing  that  a  standard  Lanczos  step  is  safe.  Observe  that 
r.^1  and  s*  are  not  computed. 

Let  us  write  R^.  =  (r^.  ,r^._^^ ) ,  S^.  =  (s^-,s.^^). 

Then  the  look-ahead  part  of  the  algorithm  comprises  the  computation  of  > 

★  *  ★ 

s^^^  and  the  unknown  elements  of  S^.R,.  ,  R^R^  ,  and  S^S^.  .  Before  specifying 
the  algorithm  we  describe  the  bumpy  tridiagonal  matrix  J  . 


the  Projection  of 


The  standard  (biorthogonal )  Lanczos  algorithm  produced  a  tridiagonal 
matrix  J.  by  the  end  of  step  j  .  The  look-ahead  algorithm  produced  a  block 

si 

tridiagonal  matrix,  also  called  J.  ,  and  written  as 

0 
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The  diagonal  blocks  are  capital  a's  and  the  subdiagonal  blocks  are  capital 
B's  .  The  A^.  are  either  1  by  1  or  2  by  2  and  the  and  are 
shaped  conformably.  For  convenience  in  finding  eigenvalues  of  J  we  have 
forced  each  B^.  to  have  one  of  the  following  four  forms 

C+]  .  [0+]  »  l^o]  ’  [o  0  ’ 

where  +  stands  for  a  positive  quantity.  It  turns  out  that  each  also 
has  rank  one. 

The  left  and  right  Lanczos  vectors  will  be  grouped  by  step  and  written 
as  P-]*  >  P2*  ’  •••  ’  Ql»  •••  »  Qj  where  sometimes  is  n  by  1 

and  sometimes  n  by  2  .  We  collect  the  vectors  together  in  Q  =  (Q,  ,  ...  ,  Q.) 

'  J 

^  /N  A 

and  P-j  =  (P-i  I  •••  >  P,-)  .  The  matrix  Q.-P^*  is  the  projector  matrix  onto 
the  left  and  right  Krylov  subspaces  and  B's  (oblique)  projection  iiito  tnem 
is  defined  by 


(Q/>)  B  (Q.Pj.)  =  q.OjP..  . 

Thus  Jj  is  the  representation  of  this  projection  with  respect  to  the  bases 
given  by  the  rows  of  P.*  and  the  columns  of  Q.  . 

V  J 

Please  not  that  j  is  not  the  order  of  J.  . 

J 


6.  The  Look-ahead  Lanczos  Algorithm  (called  LAL  hereafter) 

We  have  chosen  our  notation  to  camouflage  the  complications  caused  by 
the  fact  that  each  step  may  be  either  a  single  one  or  a  double  one.  It  turns 
out  that  quantities  are  computed  in  a  somewhat  different  order  and  way  from 
the  standard  two  sided  Lanczos  algorithm  (called  LAN)  and  the  reader  may 


1 
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find  the  differences  loom  larger  than  the  similarities.  We  have  found  it 
helpful  to  think  that  step  i  takes  certain  residual  matrices  R^-  and  , 
decides  how  to  modify  them,  then  turns  them  into  the  new  and  P^-*  ,  and 

finally  computes  part  of  the  next  set  of  residual  matrices. 

It  is  convenient  to  define  the  index  1  by 

a  =  1  +  order  (Q^_^ )  . 

Thus  by  the  end  of  step  i  we  shall  have 

,  if  step  i  is  single, 

‘  double  . 

Similarly  for  P>  .  However,  in  all  cases, 

In  describing  LAL  we  call  any  computation  involving  n  scalar  multi¬ 
plications  or  divisions  a  vector  operation  and  abbreviate  it  as  1  v.op.  . 
The  algorithm  requires  that  the  user  supply  a  subroutine  (or  subroutines) 
for  computing  Bx  and  y*B  from  x  and  y*  .  The  cost  of  these  operations 
will  be  problem  dependent.  We  stress  that  this  is  the  only  way  in  which  B 
enters  the  process. 

Step  i  of  LAL 

On  hand  are  P.j_*  ,  Q.j_1  (both  are  null  when  i  -  1)  ,  and  z.  which  is 
a  multiple  of  column  1  of  r^-  (zi  =  1)  ,  together  with  non  null  residual 
vectors  r^  ,  s*  and  scalars  o)  =  =  s*r^  ,  |1  |i  ,  ||  s*  ||  . 


(a)  Complete  R^.  =  )  and  S|  = 


"t+i  ■  -  Oi-i'i"  • 


h*‘ '  -  “^1-* 


(2  matrix-vector  products  +  2  or  3  vector  ops.) 


(b)  Compute  needed  inner  products. 


®  ■  ^£’"£+1  “  ^£+l'"«,  ,  '  ^£+*’^£+1  ’ 


'*£'"£+1  *  ^£^£+1  ’  'I  '^£+1 


.  I  s, 


(6  V.  ops. ) 


(c)  Compute  cosines  of  important  angles. 


0^  =  cos  ,  s^)  =  u)  /j|  r^^ll  •  jj  s^^K  ,  4>2  =  0 


if  9=0  then  go  to  step  2,  otherwise 


T.|  =  0)  /  9  ,  T2  =  oj  /  0  . 

II  fi+ill  ■  •'(II  •'will  ^  -  2i2('-tVi)  ♦  ^zll  ''ill 
lUjII  '  /(II  Sjll  ^  -  2t,(s*Sj^,)  ♦  r\  II  11  2) 

=  coSi{r^,  s*^^)  =  0/11  rjjll  -il  s*^^!l  , 

^2  =  cos^CPj^^^.s*)  =  ((DTj  -  0)  /  II  rj^+ill  'll  II  . 

<(>2  =  min  {|(<;i|  ,|!;>2l^  • 

(0  v.ops. ) 
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Single 


“i 

(0  V.  ops. ) 


^1+}  ■  'i+l 

c*  s 

^i+1  «.+! 

(2  V.  ops.) 


Double 

complete 

"i  •  ^i 

(0  Sj) 

Bi=  ( 

(0  V.  1 

form  new 

residuals 

/  , 

""i+l 

/  Yj,  . 

^i+2  = 

.-1 


or 


c  :■) 


=  (0  1)  (Pt  B  -  B.P^_f)  , 

(2  matrix-vector  products 
+  2  or  3  V.  ops. ) 


(e)  form 


aj  =  1/T, 


(0  V.  ops. ) 


A. 

n 


^2 


Pi 


Pi 


(1  V.  op.) 


r 


(f)  orthogonal ize 


Double 


(2  V.  ops. ) 


Vl  ^  ^+2  '  '^i^\l/’ 
5^.2  *  H*2  -  <’  “>  *i'T  • 

(4  V.  ops.) 


(g)  inner  products  for  next  step 


+  rJ|  compute 


'■  ^.l+l 


-  2a,s*s.^^  +  S5JI 


1+2  ■'  ’ 


'^i.+  l  •*■-../ 


(0  V.  ops. ) 


(h)  reset  z 


'^Z+2  ^  ^2.+2  1+2 


(3  V.  ops. ) 


"w  ‘  I” 


'i+l  y-uj  •  1^2  ^i+l  ^ 


end  of  step  i  of  LAL. 


Total  Cost  of  step  i  . 


Look-ahead:  2  matrix-vector  prods  +  9  v.  ops. 

Single  step:  6  v.  ops. 

Double  step;  2  matrix-vector  prods  +  16  v.  ops. 

For  comparison  we  note  that  the  standard  two  sided  Lanczos  algorithm  which 
keeps  II  pj  1]  =  II  H  requires  2  matrix-vectors  prods  +  10  v.  ops. 

There  are  3  different  ways  of  advancing  two  steps 

LAN  :  4  matrix-vector  p's  +  20  v.  ops., 

LAL,  1  double  step  :  "  +25  v.  ops., 

LAL,  2  single  steps;  "  +  30  v.  ops. 

The  bias  factor  in  Step  3  is  a  programming  device  which  permits  LAL 
to  implement  standard  Lanczos  (bias  =  0)  or  a  sequence  of  double  steps 
(bias  =  «>).  We  do  not  claim  that  our  setting  (bias  =  1)  is  optimal,  but  we 
doubt  that  it  is  far  off. 

7.  Numerical  Examples 

We  give  a  few  examples  which  contrast  the  performance  of  the  standard 
two  sided  Lanczos  algorithm  and  our  lookahead  variant.  They  illustruate 
the  stabilizing  effect  of  the  new  variant. 

As  mentioned  earlier  there  are  several  aspects  of  the  problem  which  we 
have  not  yet  addressed.  The  most  important  is  the  efficient  computation  of 
eigenvalues  of  J  .  All  our  computations  were  done  on  the  NOVA  840  of  the 
Remote  Sensing  Research  Program  at  the  University  of  California,  Berkeley. 
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This  forced  us  to  use  fairly  small  examples  and  this  permitted  us  to  use 
the  QR  programs  from  EISPACK  to  determine  J's  eigenvalues. 

The  Look-ahead  algorithm  reduces  to  the  regular  algorithm  when  the  bias 
factor  =  0  .  We  kept  the  bias  =  1  for  all  our  examples. 

Explanation  of  the  Tables 


Each  table  gives  snapshots  of  a  Lanczos  run,  exhibiting  what  we  feel  is 
the  essential  information. 

I  -  the  order  of  the  J-matrix. 

,(!)2  -  cosines  of  angles  between  various  candidates  for  p*  and 
.  See  section  4  for  precise  definition. 
p(J)  -  the  spectral  radius  of  J  . 

The  goal  of  our  algorithm  is  to  keep  p-j  from  sudden  plunges  towards  0  . 

We  could  have  used  ||  J||  instead  of  p(J)  ,  as  an  indication  of  "stability". 

We  fear  the  appearance  of  arbitrarily  large  "spurious"  eigenvalues  in  J  . 

We  expect  some  of  J's  eigenvalues  to  stabilize,  as  l  increases,  at  certain 
points  in  the  complex  plane.  These  points  should  be  part  of  B's  spectrum. 

If  a  double  step  occurs  in  the  Lookahead  algorithm  for  a  particular  value 
of  I  then  (j)-]  and  are  not  defined  at  S.  +  1  and  such  places  are  indicated 
by  dashes. 

0  0  0  0  0  l' 

1  0  0  0  0  0 

0  1  0  0  0  0 

0  0  1  0  0  0 

0  0  0  1  0  0 

0  0  0  0  1  0 

r^  =  s-j  =  [1,2,3,4,5,61* 


Example  1: 


B  = 


no.  of  steps  =  6 
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The  eigenvalues  of  B  are  the  sixth  roots  of  unity.  The  size  of  the  matrix 
allows  for  the  complete  history. 

TABLE  I 


Reg. 

Lanczos 

Look- 

•ahead  Lanczos 

i 

"l 

$2 

p(J) 

02 

a 

1.000 

.1277 

.8351 

1.000 

.1277 

.8351 

.1281 

.0077 

1.503 

.1281 

.0077 

1.503 

.0072 

10-6 

1.012 

.0072 

10-^ 

1.012 

M 

10-6 

10-^ 

10-6 

.0488 

•  O^I'Oo 

10 

I-*- 

O 

tQ-^ 

4.781 

5 

6 

7 

lU 

.0068 

.0068 

1.000 

Comment: 

Steps  1  -  3  of  both  Regular  and  Look-ahead  Lanczos  are  identical.  At  step  4, 

-3 

Regular  Lanczos  normalizes  s^  and  r^  by  factors  of  10  ,  producing  elements 

in  J  of  size  10^.  Step  5  of  Regular  Lanczos  is  then  aborted  because  the 
vectors  are  now  orthogonal  to  working  precision. 

The  Look-ahead  algorithm  avoids  the  large  element  growth  in  J,  by  doing  a 
double  step.  The  resulting  J-matrix  had  eigenvalues  identical  to  B  to 
working  accuracy. 

II.  B  is  12  X  12  block  upper  triangular  with  diagonal  blocks  shown  below. 

The  upper  elements  b.^  were  randomly  distributed  in  [-5,  5]. 

'  J 
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GO 

1  ^ 
ro  tn 

2' 

95  J 

“2  -  [  ' 

^  L95 

-95  ■ 

2. 

B3  =  99  ,  B4  =  Bg  =  98  , 

^6  = 

'25 

.50 

-50' 

25J  , 

CM 

1 

0 

M 

CO 

R  -  [50 

■  [50 

1 

cn  cn 

0  0 

1  1 

=  [1,  ...  ,1]* 

No.  of  steps  taken  =  24. 


TABLE  II 


Reg. 

Lanczos 

Look- 

■ahead  Lanczos 

i 

^2 

o(J) 

"2 

^(J) 

2 

.2075 

.6644 

269.3 

.2075 

,664<+ 

1 

3 

.2101 

.2059 

83.95 

— 

— 

83.95 

.7592 

.4422 

108.5 

.7594 

.4421 

108.5 

5 

.4405 

.4718 

96.67 

.4403 

.4717 

6 

.3438 

.1787 

98.45 

- - 

— 

98.45 

7 

.1304 

.0914 

98.40 

.1304 

.0914 

98.40 

8 

.1217 

.0641 

98.57 

.1217 

.0651 

98.57 

19 

.0012 

.0389 

0 

0 

.1386 

.0373 

107.2 

20 

.0012 

.0389 

10^2 

.2115 

.2661 

— 

21 

.0012 

.0389 

10I3 

— 

164.5 

22 

.0012 

.0389 

10^^ 

.1707 

.0109 

122.0 

23 

.0012 

.0389 

10^^ 

.0164 

.0706 

— 

24 

.0012 

.0389 

•-* 

0 

00 

— 

— 

164.5 

ik.  V 


Comment  on  Example  II. 


B  is  fairly  far  from  normal.  After  24  steps,  7  of  J's  eigenvalues 
had  stabilized  to  between  3  and  7  decimal  places  using  the  Look-ahead 
algorithm.  What  is  surprising  is  that  the  regular  Lanczos  algorithm  pro¬ 
duced  J's  for  which  6  eigenvalues  had  stabilized  to  3  or  more  decimal 

18 

places  despite  elements  in  J  as  large  as  10  . 


III.  B  is  a  15  by  15  upper  triangular  matrix  with  evenly  spaced  real 

eigenvalues.  The  super  diagonal  elements  were  random  in  the  range  of 
[-10,  10].  The  diagonal  elements  were 


lOj 

0 

lOj 


75  ,  j  =  1,  ...  ,7, 

.  j  =  8, 

85  ,  j  =  9,  .. .  ,15  . 


r^j  —  s.j  —  (1,1,  ...  ,1)*. 


TABLE  III 


Reg. 

Lanczos 

^1 

4>2 

p{J) 

.3205 

.0580 

64.87 

.0646 

.2167 

154.2 

.0539 

.0535 

65.02 

.2843 

.2336 

65.00 

.1651 

.2007 

65.00 

.1516 

,0447 

65.00 

,0443 

.193^ 

134.2 

.0391 

,0303 

160.0 

.0183 

,0262 

201.8 

.0438 

.0016 

249,1 

.0070 

.0896 

676.2 

,0072 

.0068 

281.8 

.0368 

.0176 

279.0 

.0269 

.0306 

290.6 

.0303 

.1304 

332.6 

Look-ahead  Lanczos 


.3205  .0580  64,87 

.0646  .2167  - 

- - 65.02 

.2839  .2353  65.00 

. 1659  . 1990  - 

-----  65.00 

.1439  .1563  - 

-  65.00 

.3650  .1736  65.00 

.2426  .4273  - 

-  65.00 

.1691  .2296  - 

-  71,40 

.0704  ,0284  65,00 

.0518  ,0114  111.7 


Comment:  The  run  was  halted  after  30  steps. 

LAL:  To  14  eigenvalues  of  B  there  were  approximations  good  to  between 
4  and  7  decimals,  including  duplicates  of  the  two  extremes. 


LAN:  To  4  eigenvalues  of  B  there  were  approximations  good  to  between 
4  and  7  decimals. 
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IV.  100  X  100  diagonal  matrix 

diag  (B)=  (1 ,2, ... ,20,41 ,62,. .. ,440,481 ,522,. .. ,1260,1321 ,1382,. . . 
2480, 2561 ,2642, ...,4100). 

,s|  random  but  unequal 


TABLE  IV. 


Reg. 

Lanczos 

Look- 

-ahead  Lanczos 

z 

^1 

^2 

P(J) 

<}>1 

^2 

p(0) 

u 

.0633 

.0037 

3546. 

.0633 

.0037 

3546. 

5 

.0036 

.0163 

10^ 

.0036 

.0163 

6 

.0035 

.0035 

4648. 

4669. 

7 

.0541 

.0493 

4489. 

.0521 

.0460 

4303. 

8 

.0536 

.0187 

4126. 

.0626 

.0444 

4219. 

9 

.0806 

.0625 

4866 

.0519 

.0027 

4129. 

10 

.0770 

.0582 

10^ 

.0029 

.0187 

— 

11 

.0823 

.0782 

lo'* 

4062. 

12 

.0815 

.0801 

10^ 

.0052 

.0158 

13 

.0821 

.0819 

10^ 

4180. 

14 

.0819 

.0818 

10^ 

.0478 

.0367 

4180. 

15 

.0820 

.0820 

10^ 

.0425 

.0204 

4099. 

16 

.0820 

.0820 

10^ 

.0225 

.0091 

4100. 

17 

.0820 

.0820 

lo'" 

.0156 

.0042 

4102. 

18 

.0820 

.0820 

10^ 

.0089 

.0096 

— 

19 

.0820 

.0820 

10® 

4100. 

30  > 


Comment: 

After  19  steps  4  eigenvalues  of  B  were  represented  in  J  to  3  or  more 
decimal  places.  The  standard  Lanczos  lost  stability  and  no  eigenvalues  were 
represented  in  its  J  . 


8.  Conclusion 


The  Look-ahead  algorithm  is  more  complicated  than  the  standard  two-sided 
Lanczos  process.  In  return  it  seems  to  prevent  the  serious  breakdown  which 
afflicts  the  standard  program.  However,  much  more  testing  is  needed. 

Other  research  topics  which  are  relevant  to  a  satisfactory  solution  of 
the  eigenvalue  problem  for  large  nonnormal  matrices  are: 

(i)  the  use  of  reorthogonalization,  or  selective  orthogonalization. 
to  maintain  bi -orthogonality. 

(ii)  the  use  of  the  Hyman- Lag uerre  method  for  finding  eigenvalues  of  J. 

(iii)  the  use  of  the  LR  algorithm  with  interchanges  for  finding  eigen¬ 
values  of  J  . 

(iv)  incorporation  of  residual  error  bounds  as  termination  criteria. 

Saad  studies  alternative  techniques  which  produce  banded  Hessenberg 
J-matrices  in  [8]  and  he  gives  a  theoretical  treatment  of  the  standard 
two-sided  Lanczos  algorithm  in  [91. 
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